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ABSTRACT 

Performance-based policy in postsecondary vocational 
education and employment training programs has two ma^or goals. 
First, it seeks to increase the basic and job-specific skills needed 
by program participants to perform effectively in occupations related 
to training. Second, it aims to stimulate debate over what the 
appropriate outcomes for vocational education and employment training 
programs are, while freeing program providers to determine how such 
aims can best be achieved. For performance-based policy to be 
effective, it must meet four requirements: (1) it must be possible to 
define desired outcomes clearly; (2) it must be possible to measure 
these outcomes accurately and efficiently; (3) measures of 
performance must affect levels of funding; and (4) useful information 
on outcomes must be available for '::on3umers and policymakers. 
Research has demonstrated that it is possible to meet these four 
requirements. Policy implications for crafting a performance-based 
approach to vocational education and employment training policy 
include the following: (1) it is important to restate the primary 
goal of adopting a performance-based orienraticn; (2) state policy 
should require the adoption of multiple definitions of perf ormiance; 
and (3) in .'.inking funding tc performance, the state needs to 
consider carefully the differences between em.ployment training and 
vocational education programs, with performance contracting more 
appropriate for employment training. (KC) 
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PERFORMANCE-BASED POLICY OPTIONS FOR 
POSTSECONDARY VOCATIONAL EDUCATION AND 
EMPLOYMENT TRAINING PROGRAMS 



Historically, education policy in the United States has been more concerned with 
prv .ess than w* outcomes. The federal government flirted briefly with performance- 
based policy in 1970 when the Office of Economic Oppormnity launched an experiment 
aimed at determining whether private contractors could teach students more effectively than 
public schools. Although the experiment was perhaps unfairly maligned, most analysts of 
the results concluded that it was unsuccessful, and cfforf. to make federal education policy 
more outcome-oriented abruptly ceased (Campbell & Lorion, 1972; Gramlich & Koshel, 
1975). In the wake of the recent school reform movement, a few states have developed 
performance- based features for state education policies. Tennessee, for example, uses per- 
formance-based funding in its higner education system. California has tried financial in- 
centives for improving student achievement test scores. Florida uses placement standards 
in its funding for secondary vocational education; and Arizona has collected and published 
data on the effectiveness of its community colleges. Several states have explored different 
approaches to merit pay and other types of teacher or school incentive systems. 
Nevertheless, most states have stuck with procedural policies for directing public 
education. 

In the employment training field, public policy has paid more attention to out- 
comes. The Job Training Partnership Act (JTPA) has sought to revitalize the federal em- 
ployment training effort by emphasizing performance, and several states have modeled state 
effons around outcome measures and performance contracts. California's Employment 
Training Panel (ETP) programs, for example, rely heavily on performance contracting to 
achieve state policy objectives. Such efforts, however, are relatively new and untested. 
Moreover, they have tended to define performance in ways that are rather nanow and 
short-term. 

This paper explores some of the major issues surrounding the adoption of perfor- 
mance-based policies in postsecondary vocational education and employment training pro- 
grams. Performance-based policy has two major goals. First, it seeks to increase the basic 
and job specific skills needed by program participants to perform effectively in occupations 
related to training. Second, it aims to stimulate debate over what the appropriate outcomes 
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for vocational education and employment training programs are, while freeing program 
providers to determine how such aims can be best richieved. 

In thinking about the forms that such policy might take, it is imponant to distin- 
guish employment training from vocational education. By employment training, we mean 
relatively short-term programs (under twelve months) typically aimed at a particular indus- 
try or specific occupation, stressing the acquisition of job specific skills. In contrast, voca- 
tional education refers to programs that arc generally longer term (one or two years), which 
stress theoretical and conceptual principles, and arc more often targeted on clusters of oc- 
cupations. Employment training, ihereforc, places more emphasis on achieving specific la- 
bor market outcomes, while vocational education is more likely to emphasize learning out- 
comes that may be generalized across a cluster of occupations. This distinction is not iron 
clad. Employment training does not ignore learning outcomes, nor can vocational educa- 
tion be inattentive to outcomes in the labor market. Rather, it is a matter of emphasis that 
needs to be remembered as performance based policies arc developed for each set of 
programs. 

Successfully crafting performance-based policy depends on satisfying four re- 
quirements. First, it must be possible to define appropriate, multiple measures of perfor- 
mance for vocational education and employment training. Second, accurate, timely data 
must be available for these measures at reasonable cost. Third, performance on these mea- 
sures must be tied to funding. And fourth, information on the performance of various 
providers must be available to consumers and policymakers. 

The remainder of this paper explores these four requirements. First, it addresses 
the definition of performance for vocational education and employment training. Second, it 
explores alternatives for tying performance to funding and so-ategies for information disclo- 
sure. Third, it discusses the availability of data on performance and how existing data 
sources might be improved. Finally, it oners some concluding observations and 
recommendations. 



DEFINING AND MEASURING PERFORMANCE 



Traditionally, public policy has relied on labor market outcomes as primary indica- 
tors of the effectiveness of vocational education and employment training programs. Such 
measures as placement rates, time to placement, duration of employment, and earnings 
have figured prominently in assessments of these programs. Vocational educators, and to a 
lesser degree providers of employment training, have often objected to the exclusive focus 
on labor market outcomes. A variety of factors beyond their control affect labor market 
outcomes. Hence, they argue, holding them accountable for employability rather than em- 
ployment would be more appropriate forjudging program success. Employability means 
having the necessary skills to perform effectively on uie job, regardless of whether em- 
ployment opponunities are immediately available. Consequently, assessing employability 
requires attention to " set of learning outcomes, for which service providers can assume 
greater responsibility. 

Neither set of measures is sufficient for evaluating vocational education or employ- 
ment training programs. It is easy to imagine that in a flourishing economy placement rates 
and other measures of labor market outcomes could be high for even poor programs. 
Similarly, programs that teach skills that are no longer needed in the labor market perform 
little useful service, no matter how effectively such skills are taught. Consequently, a 
combination of learning and labor market outcomes are needed to evaluate program effec- 
tiveness. How should these be defined and measured? What arc some important caveats on 
their use in public policy? 



Labor Market Outcomes 

or the two sets of outcomes, labor market outcomes are probably the more familiar. 
California's Employment Training Panel (ETP) programs, for example, require placement 
and remaining on the job for ninety days as conditions for payment under the performance 
conu^acts negotiated with its O^iners. Placement rates figure prominently in JTPA, and a 
few states have established placement criteria as the primary determinant of whether to con- 
tinue specific vocational education offerings. 
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Generally, labor market outcomes include the following types of measures: 
Placement 

Placement rates, expressed as a percentage of those completing a program or as a 
percentage of those initially enrolled, have been popular measures of labor market 
outcomes. Further distinctions are often made between placement in jobs related to 
training versus placement in unrelated jobs. Additional distinctions arc sometimes 
made between public and private sector cnploymenL 

Time to Placement 

The amount of time — measured in days, weeks, or montlis — between completion 
of a program and placement on the job is sometimes used as an indicator of pro- 
gram effectiveness. Such a measure is equivalent to time unemployed between 
completion and initial placement. A related measure is time unemployed between 
jobs once an initial placement has been made and subsequently terminated. 

Duration of Employment 

Once placed, the amount of time an individual remains employed may be used as an 
indicator of program effectiveness. This may be measured as the amount of time 
spent in the initial job, amount of time with the initial employer, or amount of con- 
tinuous employment in a series of jobs or with a series of employers. 

Earnings 

Earnings arc another indicator of program effectiveness. They may be measured as 
an hourly rate at time of placement or total earnings over a month, quarter, or year. 
If collected over time, earnings are an indirect indicator of upward job mobility and 
advancement. In the event of pan-Ome employment, it is desirable to determine 
whetlier employment pan-time is voK ntary (i.e., full-time employment is not de- 
sired by the employee) or involuntary (i.e., full-time employment is desired but 
cannot be obtained). 

Employer/Employee Satisfaction 

Measures of labor market outcomes have also included indicators of satisfaction on 
the part of employers or employees. WTiile more subjective than other measures, 
these can still be quantified to some extent. 



Most of these measures of labor market outcomes do not pose serious problems in 
regards to definition or measurement. As will be discussed in more detail in the section 
entitled, "Improving State-Level Information on Performance," there are methodological 
problems in data collection that have made it difficult to obtain accurate data on labor market 
outcomes on a sufficiently representative group of program completers or to obtain data 
over periods of time that were sufficiently long enough. These requirements for improvec* 
response rates in follow-up studies and better longitudinal data will also be addressed in the 
above mentioned section. 

The major difficulty in using labor market outcomes to assess proj^am effective- 
ness lies in isolating outcomes that may truly be attributed to the effectiveness of a program 
from those that result from factors beyond the control of a program. Two obvious external 
factors are general economic conditions and student characteristics. If completers of differ- 
ent programs face different economic conditions, comparisf»ni of unadjusted labor market 
outcomes will not yield accurate comparisons of the relative effectiveness of the programs. 
Similarly, programs that serve students with different characteristics may realize different 
labor market outcomes that have littie to do with program effectiveness. For example, a 
trade program that serves mainly women may have lower placement rates than a similar 
program that serves mostly men. This is not because the program is inferior, but because 
its graduates continue to face discrimination in the labor market. Evaluations of program 
effectiveness in terms of labor market outcomes, therefore, must distinguish programmatic 
effects from outcomes that result from factors beyond the control of the vocational educa- 
tion or employment training program. The inclusion of good statistical controls will help to 
achieve this distinction. 

There is a third m^jor external variable that can affect labor market outcomes — 
namely, the knowledge and abilities participants bring to a program at the outset. It is a 
well known fact in education circles that tlie easiest way to achieve good resjits is to admit 
only good students. Controlling for these variables, however, requires measures of learn- 
ing outcomes. 
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Learning Outcomes 



Learning outcomes provide a more direct measure of program effectiveness than la- 
bor market outcomes and are less subject to external influences beyond the control of pro- 
gram providers. We have found it useful to distinguish among four different types of 
learning outcomes: (1) program completion, (2) benchmark mastery based on competency 
testing, (3) value-added, and (4) program improvement over time. As these may be less 
familiar measures of outcomes, we will elaborate on each briefly. 

Program Completion 

Program completion takes many forms in vocational education and employment 
training programs. It can mean the awarding of a formal degree or certificate, the complet- 
ing of an integrated sequence of courses or other plan of study, or simply fmishing a single 
course designed to meet a very immediate and specific skill need. In many instances, 
completion depends upon accumulating a certain number of credits with satisfactory 
grades. In some cases, it may mean little more than simply having put in a certain amount 
c? time attending classes and participating in laboratories. 

Alternatively, in contrast to credit accumulation or passing specified amounts of 
time, completion can represent the mastery of a certain set of skills — that is, demonstrated 
acquisition of the general and job specific skills necessary to perform effectively an entry 
level job in an occupation related to the student's training. In this case, the critical test is 
mastery, regardless of ths amount of time mastery requires. For some students, mastery 
may take a long time; for others, it may be accomplished quite quickly. In every case, 
however, a program participant is not considered to have completed the program until suc- 
cessful mastery of the required skills has been accomplished. 

Note the emphasis on "demonstrated acquisition." Passing time in vocational edu- 
cation or receiving a degree or certificate count for nothing under such a definition unless 
students have demonstrated in some measurable fashion that the necessary general and job 
specific skills have b**,en acquired. Such demonstration, therefore, depends on satisfying 
three requirements. First, it must be possible to determine rather precisely the kinds of 
general and job specific skills required to perform a particular job. Second, it mu.st be 
possible to specify the degree of mastery of these skills that is likely to be needed for satis- 
factory performance on the job. Third, it must be possible to measure objectively whether 



necessary levels of mastery have been achieved. Competency testing is one method for 
meeting these thiw requirements. 

Although competency-based cuiricula and competency testing are widely used in 
vocational education, they have never been recognized as possible tools for educational 
policy. Rather, policy experience with competencies comes mainly from the Job Training 
Partnership Act (JTPA), as well as some state employment training programs. 

The term "competency" r ;fers to leamablc outcomes. Examples of different kinds 
of competencies include skills such as typing or computer programming; knowledge such 
as an awareness of career options or academic knowledge; attitudes such as regard for 
others or initiative; and behavior such as prompmess, cooperation, or dress standards 
(Center for Employment and Income Smdies, n.d.). Competency-based learning programs 
structure curricula by identifying competencies which students need to learn, building cur- 
ricula materials to teach these specific competencies, testing students on their mastery of the 
skills, and advancing smdents through the program as they gain competencies. 

Curricula based on competencies can create improved learning environments for 
students. In a study of the JTPA Youth Competency-based Employment and Training 
Program, The Center for Employment and Income Smdies (n.d., p. 9) identified six advan- 
tages to the competency-based approach. First, a competency-based approach can readily 
identify and address individual needs. Because assessment is targeted to the individual and 
because ser/ices may be provided which specifically address skill needs as revealed by an 
assessment, individual needs are consistently the focus of program activities. Second, this 
approach fosters achievement by providing specific, attainable objectives which can be 
measured in terms that are clear both to the participant and to program staff. Third, a com- 
petency-based approach frequently increases motivation and interest in participation in ser- 
vices. Youth (and most adults as well) tend to respond positively to situations in which 
learning is determined by demonstration of a skill, rather than by length of time one has 
been in a classroom or work situation. Fourth, a competency-based approach provides for 
a more realistic understanding of one's own skills, abilities, and preferences, since it con- 
centTLtes on what a person actually does. Fifth, because competency-based approaches 
imply objective standards of achievement, they tend to have credibility both for youth and 
the community. Youth tend to feel that they are not victims of the whim of an unfriendly 
teacher or counselor. Finally, competency-based approaches that are based on employer 
input tend to offer the individual a more realistic picture of the world of work and the job 
market. 



Although offering powerful features, programs that try to use competency-based 
curricula must address some important issues. First, either program administrators or the 
training agency must decide which competencies programs will teach. This decision in- 
volves setting priorities among skills and choosing skills that arc appropriate for the occu- 
pation taught. Second, there are m?ny practical problems. Implementation of competency- 
based curriculum requires that definitions of competence be reduced to mwiageable terms, 
broken into recognizable and teachable units, and assessed reliably. Dangers include 
breaking the cuniculum into specific measurable skills which do not sum up to the re- 
quirements of the occupation. Third, competency-based programs create assessment is- 
sues. Programs must decide how to measure student success. If all skills needed for an 
occupation are not defined in the competencies, the measure of students* success — instead 
of the demands of the occupation — can dictate what is taught (i.e., teaching to the test). A 
fourth problem is setting standards or benchmarks of success. If standards arc too low 
they will erode motivation to excel, yet if they arc set too high, too few people pass the 
course. In addition, standards may be set in order to pass a certain percentage of students, 
while actual job success would dictate a different standard. 

Benchmark Mastery 

Benchmark mastery is a subset of competency-based program completion. By 
definition, completion means mastery of the full complement of skills necessary for effec- 
tive perforaiance on the job. Mastery of some but not all of the skills, however, represents 
learning. By assessing the mastery of individual skills as a program progresses, rather 
than waiting until the end, it is possible to keep track of partial completion and the effec- 
tiveness of various components of a full program. For example, if a program consists of 
three or four separate courses, monitoring the achievements of students in each course will 
provide a means for evaluating the effectiveness of each course, even though mastery of the 
material in any individual course is insufficient for successful performance on the job. 

Measuring mastery at the end of a program or at the end of its components, how- 
ever, does not necessarily indicate that learning has occurred unless we assume that all 
students started with no knowledge. Measuring competencies only at the end of a program 
may also encourage institutions to enroll students who have the least to learn (see 
Chickering & Claxton, 1981; Micek & Amey. 1973). Indicators of the gain in knowledge, 
or "value-added," are therefore desirable. 
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Value-Added 

'.vhile competency-based program completion constitutes a good indicator of a de- 
sirable outcome for vocational education programs, it is not a sufficient indicator of pro- 
gram effectiveness at imparting general and job specific skills. Students entering a program 
with high levels of mastery may complete with little or no assistance from the program. 
Moreover, programs may be tempted to cream in order to obtain higher completion rates. 
A better indicator of program effectiveness, therefore, is 'Value-added" — the amount of 
general and job specific skills acquired between entry to and exit from the vocational edu- 
cation program. 

Competency-based curricula and competency testing lend themselves well to 
measures of value-added, provided assessment is done at the beginning as well as at the 
end of the program. There are, however, potential problems that need to be addressed. 
First, it is sometimes difficult to determine precisely what has been responsible for what- 
ever value-added is observed. For example, suppose that value-added testing reveals sig- 
nificant gains in the math and science skills required for a panicular occupation while 
participating in a vocational education program. If participation was limited to the 
vocational education program, these gains are rightiy attributed to the program. However, 
if the student also participated in math and sciences courses outside the vocational education 
program, it will be difficult to sort out precisely what aspects of the postsecondary 
curriculum were responsible for the gains. 

Second, while value-added indicators of performance are less likely to encourage 
creaming (because programs get credit for what they impart instead of for what students 
bring), one must question die value of gains that fall short of program completion. If, by 
definition, completion represents the acceptable standard for entry-level employment, then 
anything short of completion is not likely to lead to employment. A special needs student 
may make great gains in a vocational education program on a value-added measure, but if 
the student fails to complete, what really has been accomplished? Successful special educa- 
tion programs have addressed this problem not by accepting subcompletion accomplish- 
ment but by working with employers to redefine job responsibilities. They aim to define a 
job commensurate with the abilities of a handicapped student to perform it. In effect, they 
create a new kind of entry-level job and design standards of program completion that are 
appropriate for it. Students must still complete the occupational program, therefore, to per- 
form effectively in this redefined job. In the final analysis, value-added is not a substitute 



for program completion. Rather, it is an additional indicator of program effectiveness that 
should be examined along with completion. 

Program Improvement Over Time 

Value-added measures the learning gains of individual participants over time. It is 
also sometimes useful to measure changes over time in the accomplishments of courses, 
programs, or institutions. Such measures are particularly important when performance is 
tied to funding. Funding procedures that reward only the high achieven may ignore pro- 
grams that have realized the greatest gains. Such procedures are subject to the criticism that 
they reward programs that are least in need of additional resources. By monitoring the 
progress of initially low-achieving programs and rewarding improvement, a performance- 
based policy will encourage all programs to strive for better results. 

There is, however, a risk in such an approach. While poor programs may indeed 
realize substantial improvement (e.g., raising the average scores of their panicipants by, 
say, fifteen percent ovct the previous year), it is possible that such gains have little practical 
effect on improving the employability of the participants. The gains can be large in abso- 
lute terms, but still be well short of what participants need to work successfully in a job re- 
lated to their training. From a policy perspective, therefore, eliminating the program may 
be preferable to rewarding it 

Additional Caveats 

In contrast to labor market outcomes, we have relatively little experience with the 
systematic, widespread measurement of learning outcomes in postsccondary vocational ed- 
ucation and employment training. While competency-based curricula are used widely in 
postsecondary vocational education, there are no uniform standards that would permit 
comparisons of performance across programs or institutions. JTPA has encouraged the 
adoption of competencies and benchmarks for youth, bi t these also are far from uniform. 
Similarly, the development of competency- based approaches to employment training have 
been highly deccntralizec* and idiosyncratic. 

Assuming that some standardization is desirable for purposes of improving ac- 
countability through cross-program comparisons (and the benefits of standardization must 
be weighed against the benefits of decentralized programs individually tailored to local 
needs), it must be remembered that curriculum development, the design of assessment in- 
stmments, and the conduct of assessment arc costly and time consuming. Moreover, paper 
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and pencil tests are not always well suited to assessing the mastery of competencies. 
Testing that involves doing (for example, constructing a circuit board that works when it is 
plugged in) is often a more effective means of assessment than a standardized test, but such 
testing is more difficult to incorporate into a standardized statewide reporting system. 
Consequently, adopting performance-based policies must strategically select progranris for 
initial inclusion in a performance-oriented system. It may be wise, for example, to begin 
with programs that are subject to state licensing requirements. 

Additionally, it is important not to become enamored with any one measure of out- 
comes. Each has flaws and, if used in isolation, is likely to promote undesirable behavior. 
An exclusive emphasis on program completion, for example, may promote creaming. 
Focusing solely on value-added may divert attention from the bottom line — successful 
placement and effective performance on the job. Multiple indicators of performance will 
help to keep the system attentive to performance broadly conceived and avoid skewing it in 
one narrow direction. 

Finally, it is important to underscore once more the need for procedures for evaluat- 
ing performance in the context of external factors and student characteristics. Learning out- 
comes may vary with differences in fiscal resources available to program providers. They 
certainly will vary with differences in the abilities and needs of program participants. 

TYING PERFORMANCE TO FUNDING 

The success of performance-based policies will depend in large measure on finding 
effective ways to tie performance to funding. While better definition and measurement of 
desiiied outcomes can improve general oversight and administration of vocational education 
programs, mere exhortation to excel on these measures is not likely to achieve much result. 
Rather, f.nancial rewards for good performance, and possioly sanctions for poor perfor- 
mance, are more likely to focus atte^ition on program performance and ways to improve it. 

While tying performance to funding has a certain intuitive appeal, designing fair and 
effective performance-based funding arrangements is not easy. A numb<^r of important 
policy issues must be addressed. First, what should be the general relationship between 
funding and performance? Should resources be directed only to the stronger performers or 
should policy direct more resources to poor perfomiers to strengthen their efforts? Second, 
at what level should performance be evaluated and compared — community college districts 
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of SDAs, colleges or community-based organizations* programs within institutions, indi- 
vidual courses, or individual teachers? Regardless of how performance is measured, to 
whom should rewards be given and should any constraints be placed on the forms such 
rewards take? Third, who should administer performance-based systems — a state coordi- 
nating board, particular agencies witi?in the state, local boards or SDAs, or individual insti- 
tutions? How much discretion should the administrators of these policies have over how 
performance is defined, measured, and rewarded? Fourth, what mear.i for tying perfor- 
mance to funding — ^performance contracting or performance-based funds allocation formu- 
las — should be encouraged and under what circumstances? What procedures will be used 
tf :letermine that these procedures are designed fairly and with adequate technical expertise? 

To help answer these questions, we reviewed recent experiences with two types of 
approaches to linking performance and funding: performance contracting and performance- 
based funding formulas. Our findings will be briefly summarized here (Hoachlander, 
Choy, & Brown, 1989). 



Performance Contracting 

Although tried from time to time in education, performance contracting for 
instructional services has never become popular. In contrast, performance contracting has 
become a central feature of JTPA and other employment training programs. The California 
Employment Training Panel (FTP), charged with administering the state's fifty-five million 
dollar program for retraining displaced workers, relies entirely on performance contracts 
with commuiuty colleges, proprietary schools, and community-based organizations. The 
training agency is not paid for training a client unless the person is placed in an unsubsi- 
dized job for at least ninety days. We examined both JTPA and ETP to Icam more about 
their experiences with performance contracting. 

Performance Contracting Under JTPA 

Performance has been the watchword of JTPA from its outset. Replacing the much 
maligned Comprehensive Employment Training Act legislation ir 1982, JTPA was billed 
as a no-nonsense, outcome-driven training program for the disadvantaged — a first of its 
kind, efficientiy funded, and rehant on the increased involvement of the private sector to 
keep it focused on the bottom line. Curiously, while JTPA established a system of perfor- 
mance standards and rewards based on results, performance contracting per se was never 







mentioned in the act. Nevertheless, during the last three or four years, performance con- 
tracting has been widely adopted throughout the system and has become the primary con- 
tractual tool in many Service Delivery Areas (SDAs), the basic organizing unit of JTPA. 
To Icam more about the experience of JTPA with performance contracting, we visited sev- 
eral SDAs around the country. We also commissioned a paper and a small survey of SDAs 
to determine how widely performance contracting was used, why, and with what results. 

While JTPA established performance standards as the primary means of account- 
ability for the program, it left states and localities a great deal of discretion in implementing 
this system. SDAs were free to organize and operate their own training programs **in- 
house," or they could make arrangements with other local agencies and institutions such as 
community-based organizations, community colleges, and proprietary schools. If SDAs 
decided to contract out training, there were no requirements that they do so in any particular 
way. Regardless of how they chose to organize the training, however, the SDAs would be 
held accountable to measures of placement, welfare reduction, cost per placement, and av- 
erage wage at placement Although performance contracting is a natural extension of ac- 
countability from the SDA to its agents, it was by no means an inevitable developmeni. 
Nor was it required anywhere by law or regulations. 

It is, therefore, testimony to the appeal of performance contracting that this practice 
has swept through what, from an operational perspective, is a rather decentralized, diverse 
system. A recent survey of one-hundred SDAs in twelve states found that over eighty per- 
cent of the SDAs were using performance contracting — ^procedures that tie all or a part of 
payments for tr ining to achieving specified levels of performance on negotiated measures 
of performance (Butier, 1988). Why has performance contracting become so popular in 
JTPA? For what is it used? What are its pros and cons? 

SDA officials who have adopted performance contracting typically offer two main 
reasons. First, performance contracting is a management tool. Fifty-eight percent of the 
administrators surveyed by Erik Butier (1988) cited better accountability as one of the rea- 
sons they used performance contracting. It directiy translates the performance standards by 
which the SDA is held accountable into measures for evaluating and compensating whoever 
delivers the training services. Performance contracting, therefore, assures the SDA that its 
contractors understand and will work to achieve the same objectives that will determine the 
success or failure of the SDA. 
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Second, performance contracting under JTPA has been an ideal means for appear- 
ing to minimize administrative costs, and almost sixty percent of SDA administrators cited 
this as one of their reasons for opting for performance contracting. JTPA places a limii of 
fifteen percent on the percentage of JTPA funds that may be used for administration. SDAs 
have struggled to stay within this limit and have found it particularly difficult to achieve 
when programs arc run and administered in-house. Through performance contracting, 
SDAs have in effect been able to transfer some of the costs of program administradon to 
their contractors and buiy this in the performance contract The entire cost of the perfor- 
mance contract may be counted as training services and, therefore, is not subject to the fif- 
teen percent limit It is possible that there is some real reduction in administrative costs 
from performance contracting. By relying on existing community-based organizations, 
conununity colleges, and other local training providers, an SDA avoids recreating and du- 
plicating services and the administration of these services that must inevitably accompany 
them. It is impossible to tell, however, how much of the reduction in administration pro- 
duced by performance contracting is real rather than apparent 

Interestingly, only forty percent of the SDA administrators said that better training 
was one of the benefits of performance contracting, and only thirty-seven percent cited 
better placement — one of the primary performance measures by which the programs are 
held accountable. Only six percent cited better recruitment, which is not a surprising result 
since most of the SDAs retain major responsibility of intake and determining eligibility, 
functions that they arc in good positions to perform. From our review and site visits, how- 
ever, tiicrc is as yet littie hard evidence that performance contracting improves the quality of 
training or its effectiveness. It well may, but no one really knows. 

Over eighty percent of die SDA surveyed b> Butier said tiiey had one or more per- 
formance contracts. Seventy-three percent said they had performance contracts for class- 
room-based vocational training, fifty-three percent for pre-employmcnt/work maturity skill 
training, fifty percent for basic education, forty-eight percent for on-the-job training, and 
thirty-two percent for job search assistance. 

SDAs, then, appear to be most comfortable wiUi performance contracting when it is 
used to obtain classroom- based vocational training, probably because these kinds of ser- 
vices are the easiest to obtain through other service providers. Classroom programs in 
community colleges and community-based programs are long establi ed. It makes little 
sense for SDAs to duplicate these. 



As popular as performance contracting has become in JTPA, SDA administrators 
acknowledge that it is not free of problems. There are three frequently cited drawbacks to 
performance contracting: (1) creaming, (2) excessive concern with numbers, and (3) the 
timing and level of reimbursements. 

From the very stan of JTPA, the new emphasis on performance raised widespread 
concerns about incentives to screen out clients who were the least likely to succeed in 
training programs and to admit only those who were the most likely to enable programs to 
accomplish their performance objectives. There are undoubtedly incentives to cream, and 
there is some evidence that JTPA may overstress quick results, inexpensive placements, 
and the selection of clients most likely to benefit from training (Grinker Associates, Inc., 
1986). However, given the eligibility requirements of JTPA, concerns about creaming 
may be overemphasized. Moreover, it is not performance contracting per se that leads to 
creaming but, rather, the way the perfomiance objectives are defined. I: is easy to imagine 
how one could structure a performance contract to ensure that more difficult-to- train, 
harder-to-place clients were admitted to training programs. Performance contracting can be 
a means to many ends. 

Defining the ends, then, is the more serious issue, and on this score performance 
contracting in JTPA raises some more troublesome problems. SDA administrators cite a 
tendency to become preoccupied with the numbers, churning people through the system to 
meet placement perfomiance standards with inadequate attention to quality of programs and 
longer term benefits for participants. That JTPA places high percentages of its clients in 
jobs is indisputable. Less is known, however, about how long they last, how effectively 
they have been trained, or how generalizable the skills they acquire in JTPA are to other 
jobs they may encounter during their working lives. Moreover, with the exception of 
youth programs where competencies have figured prominentiy in JTPA, programs have 
generally ignored developing measures of value-added. This failure pardy reflects the dif- 
ficulty of devising good competency-based training curricula, but it also reflects a resis- 
tance to divert attention from "the bottom line" — employment 

Finally, perfoniiancc contracting sometimes creates high risks and serious cash 
ilow problems for agencies that are not in strong positions to cope with such strains. Since 
part or all of the payments are not received until after clients coi iplete programs and are 
placed, service providers must front a considerable amount of ex; enses. Moreover, they 
do so with considerable risk that unless they meet the performance objectives, they may not 
be paid at all. It is, of course, such prospects that lend performance contracting its power. 



that is, the costs of not performing are high and the benefits of success are great. 
Nevertheless, the all or nothing features of some performance contracting may dissuade ef- 
fective trainers from participating in JTPA and may encourage those who do participate to 
adhere mindlessly to performance criteria, cream, or otherwise dilute the quality of their 
programs.^ 

Performance Contracting in the California Employment Training Panel 

The California Legislature created the Employment Training Panel (ETP) in 1983 in re- 
sponse to concerns that the changing world economy and the introduction of new tech- 
nologies were causing workers to lose their jobs. ETP focuses on the specific needs of the 
economy and supports training and retraining that improves productivity and competitive- 
ness and promotes security of employment for California workers. Up to fifty-five million 
dollars a year can be allocated to training programs operated by employers or public or pri- 
vate training agencies. 

Several features distinguish the ETP program from other employment training pro- 
grams. First, the ETP program uses fixed fee performance contracts exclusively. ETP 
reimburses the training agency at a fixed amount per trainee if and only if the trainee suc- 
cessfully completes the training and is employed by a single employer for ninety days. 
There are no partial reimbursements for trainees who drop out of training or who do not get 
and keep jobs. 

Second, training is hnked to specific jobs rather than to general labor market needs. 
Proposals for contracts may be initiated directly by an employer or a group of employers, 
by a training agency, or by a master contractor — an organization such as the State 
Department of Education paid to provide marketing and outreach services. Proposals must 
include, among other things, a list of employers who will hire successful completers and 
signed agreements that they will participate in the development and operation of the pro- 
gram; a statement of the need for and purpose of the training; a description of the skills re- 
quired for the jobs; a schedule and plan for conducting the training; and the fee that will be 
paid. 



•Not all JTPA contracts operate on an "all or nothing" basis. Some SDAs pay contractors at different 
stages of performance. For example, ihey may pay contractors fifteen percent of their fee upon meeting 
recniiunent goals, thirty percent for retaining trainees until the the end of the program, thirty percent for 
initial placement, and twenty percent for job retention 
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Third, the program is limited to individuals who are receiving (or have recently ex- 
hausted) unemployment benefits or who are in danger of being laid off and becoming 
unemployment insurance recipients. In other words, individuals who have never been 
employed or who have not been employed long enough to earn unemployment benefits are 
not eligible. This limit on paniciparion is the result of the funding source — a surtax on 
employer payments to the unemployment insurance fund. 

For programs training unemployed workers, contractors are responsible for recruit- 
ing trainees and are free to select whomever they wish from the eligible pool. They work 
with the Employment Development Department offices to identify potential participants and 
do direct mail and newspaper advertising. For programs involving retraining of workers in 
danger of losing their jobs, the employers (and unions, sometimes) select participants from 
among their own workers. Program regulations require contractors to describe in their 
plans efforts they will make to recruit women, minorities, the disabled, and veterans. 

The FTP program started slowly. During the first two years of full operation, the 
number of individuals who had actually completed training and had been employed for 
ninety days was only 8.6 percent of what was planned (Legislative Analyst Office, 1986, 
p. 20). As a result, a large proportion of the appropriated funds remained unspent. As 
experience with the program grew and administrative deficiencies were corrected, the level 
of activity increased. In 1986-87, placements were sixty-five percent of what was planned. 
In addition, demand for FTP assistance exceeded the supply of available funds for the first 
time. The average cost per person was $2,061 (Employment Training Panel, 1987, p. 14). 

An evaluation conducted by the Training Research Corporation for the FTP con- 
cluded that the FTP program efficiently moved unemployed workers into new jobs and en- 
hanced the productivity of potentially displaced workers (Moore, Wilms, & Bolus, 1988). 
It found that the program had a positive effect on earnings and that it reduced unemploy- 
ment for participants. 

The major criticism that has been levied against the FTP program is that the ETP 
has subsidized normal employer training costs — in other words, paid for retraining of em- 
ployees who were not truly in danger of being laid off if ETP had not provided training 
funds. The Legislative Analyst's report (1986), which made this charge, came to this con- 
clusion after interviewing employers. It cited as an example an aerospace firm with a con- 
tract to train machinists in the use of computer-controlled machining equipment. Because 
there was a shortage of skilled machinists, the evaluators were very skeptical of the claim 



that the firm would have laid off its experienced machinists and attempted to hire even more 
experienced ones if they had not received ETP funds. They thought it much more likely 
that the firm would have provided training at its own expense to the machinists already in 
its employ. 

Issues of the appropriate use of public funds aside, as an experiment in perfor- 
mance contracting, the ETP program has been successful. What follows are some general 
observations about what has made perfomance contracting woxlc in the ETP program and a 
discussion of the implications for performance contracting in postsecondary vocational ed- 
ucation. These observations are based on interviews with state-level administrators of the 
ETP program and site visits to four contractors — three community colleges and one re- 
gional occupational program with a large adult education program.^ 

Performance contracting is risky, especially in the case of the ETP program where 
the criterion for success is so strict. Although it has not wori;ed well for all contractors, 
many have been able to make it work. Described below are factors that have contributed to 
ETP's being able to attract contractors and contractors being able to operate successful 
programs. 

• Clear goals a i measurable oiacomes 

The goals of the ETP program are clear. To place unemployed workers in good 
jobs with long-term career potential and to keep employed workers whose jobs are 
threatened because they lack the necessary skills in their jobs. No other outcome is 
acceptable. The ETP program is narrowly focused on teaching job specific skills 
that have been identified as important by employers who are committed to hiring (or 
retaining) the completers. Contractors are expected to teach trainees only those 
skills. Expectations arc very clearly spelled out — the proposal for funding must 
include a schedule for the training and a plan for conducting the training that de- 
scribes what will be learned and how. Contractors accept placement and retention 
on the job for ninety days as the appropriate criterion for success and are willing to 
participate on those terms. If the goals were less specific, or if there were multiple 
goals with unclear priorities, or if the measures of success were less clearly related 
to the goals, getting contractors to participate would be much harder. 



^Proprietary schools have not played an impoitani role in the ETP program, and none currently have 
programs. 



Data collection on cutcomes is easily accomplished. Contractors need only keep 
track of placements and verify with the employers that the trainees arc still em- 
ployed in the same place ninety days later. None of those interviewed found data 
collection a problem. Only very rarely did a trainee "disappear." On a random ba- 
sis, employment status is veriiied at the state level by using unemployment insur- 
ance r^ords. 

Involvement of employers 

The employers who will eventually hire the completers must be identified before the 
contract is awarded. The employers not only are allowed, but are required, to assist 
in the development and operation of the program. Employer involvement goes be- 
yond the advisory conmiittees that typically meet once or twice a year to provide 
advice on vocational education programs. In the ETP progrzun, the employer 
shares decision making authority with the training agency. With such a stnicmre, 
there should be no questions at the end of the program as to whether or not the 
trainees have the appropriate skills. This makes placement more certain and partici- 
pation much less risky for contractors. 

The training contractors that have been the most successful devote a great deal of 
time and energy to identifying employers and developing good relationships with 
them, and they believe strongly that this effort has contributed to their success. 
While some contracts involve only one large employer, many involve groups of 
smaU or medium sized employers. For example, one community college we visited 
conducted a computer-assisted design course for two-hundred architectural firms. 
Many word processing and office automation programs also involve a large number 
of small employers. 

Nature of the population served 

As indicated above, participation in the ETP program is limited to individuals who 
are receiving (or who have recently exhausted) unemployment benefits or who are 
in danger of being laid off and becoming unemployment insurance beneficiaries. In 
other words, only individuals who have been employed or who have been em- 
ployed long enough to earn unemployment benefits are ehgible. These individuals 
have already demonstrated that they have general job readiness skills — in fact, 
many of them are already employed — and are thus much lower risks on average 
than the populations typically served by job training or vocational education pro- 
grams. Program directors interviewed who had experience with JTPA training 
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programs remarked on the enormous difference in the job readiness of the popula- 
tions served. 

One community college provided an example of a "low risk" group served. The 
college developed a seven-month retraining program for workers laid off when an 
auto plant closed The program trained participants in microwave technology, and 
was successful largely because the trainees were the "cream of the crop." They 
were all in their thirties or forties, they were secure in their training situation be- 
cause they had received severance pay, they were highly motivated to find jobs, and 
most had many years of work experience. In addition, the job market for mi- 
crowave technicians was good. As a result, sixteen out of the seventeen persons 
enrolled were placed and retained in jobs for at least ninety days. 

All the contractors we visited screen prospective trainees for new-hire programs 
veiy carefully. They test them for basic reading and math skills and interview them 
to assess their ability to complete the program and their employability. Only those 
expected to succeed arc accepted, and poor performers are terminated if necessary. 
The programs cannot afford to take chances or waste resources on trainees who will 
obviously not be successful. The contractors also reject candidates with too many 
skills on the grounds that they are likely to leave the program for a job before the 
end. If this were to happen the contractor would not be paid for that person. 

Program regulations require contractors to describe in their plans what efforts they 
will make to recruit women, minorities, the disabled, and veterans. These plans arc 
evaluated during the selection process: however, once a contract has been awarded, 
the contractor chooses the participants. 

Focus on short programs 

Although the law permits training programs to last as long as eighteen months, the 
average program for retraining workers is only 234 hours, and the average program 
for the unemployed is only 503 hours (only about three months for an eight-hour 
pe- day program). One reason that contractors have preferred shorter programs is 
that it is easier to get employers to keep their comr^itments. Training agencies cited 
instances of employers who were willing at the stan to hire trainees, but who 
changed their minds before the end of the training period, forcing the agency to find 
other placements. 
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Emphasis on retraining 

Programs for retraining workers already on the job arc much less risky than pro- 
grams for the unemployed. A worker already on the job who has showed interest 
by agreeing to participate is very likely to complete the training and remain on the 
job for ninety days. The mix of training for the unemployed versus retraining has 
shifted during the life of the ETP program. Whereas fifty-five to sixty percent of 
the participants were unemployed in 19 "S. eighty-five to ninety percent of the par- 
ticipants are now workers being retrained. Of the individual training projects ap- 
proved by the ETP as of June 30, 1987, only twenty-seven percent were for the 
unemployed, compared to fifty-three percent for retraining. Twenty percent in- 
cluded both. 

The emphasis on retraining is not entirely attributable to the lower risk, however. 
Also contributing is the difficulty experienced by many contractors in recruiting 
trainees for new-hire programs. Among the causes of this problem, which has ex- 
isted since the inception of the ETP program, are low unemployment rates and the 
eligibility requirement Because only individuals who arr receiving unemployment 
insurance (or have recently exhausted their benefits) are eligible to participate, 
ETP's executive director estimates that only one-quarter to one-third of the unem- 
ployed are eligible for new-hire programs. 

Commimient of resources to placement and support services 
Each of the training agencies visited had made a heavy commitment of resources to 
placement. There was no assumption that simply teaching workers the right skills 
would guarantee them jobs. Rather, it was taken for granted that people need help 
in locating opportunities and interviewing for jobs, and full-time counselors, or 
"employment specialists," were hired to provide this help. In two of the programs, 
the contractors conducted classes in job search skills, helped trainees with their re- 
sumes, and sponsored job clubs. In addition, the employment specialists followed 
up after placement to make sure everything was going smoothly. 

One community college visited achieved a one-hundred percent placement rate for 
one new-hire contract, and claims to hi^ve the best contract results of any training 
agency in the state. The director attributes this record to the resources they invest in 
suppon services and placement They operate on the principle that the more atten- 
tion paid to students, the better the placement rate will be. Attendance at class is 
mandatory, and if a trainee misses a class, a staff person calls the next day to find 
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out why. Saturday tutorial sessions are conducted for students who have niissed 
classes or need extra help. For new-hire students, the college provides extensive 
support during the job search process, including instruction in interviev/ skills, help 
in getting interviews, and videotapes of real interviews. College resources such as 
child care services and libraries are made available to ETP participants. The director 
attaches ^rsat importance to the "personal touch," and believes that it makes the dif- 
ference between a sixty percent and one-hundred percent placement rate. 



Implications for Policy 

There is good evidence that performance contracting has worked well in a variety of 
employment training settings. It is by no means clear, however, that it could easily be ap- 
plied to postsecondary vocational education programs. There are important differences 
between JTPA/ETP programs and postsecondary vocational dducation that would make it 
difficult to institute performance contracting as formulated for JTPA or ETP. First, the 
employment training programs typically have a much narrower focus than vocational edu- 
cation. JTPA and ETP trainees arc taught mainly those skills needed to obtain or retain a 
certain job. In contrast, students in postsecondary vocational education programs have to 
meet general education requirements. They are encouraged to explore alternatives and even 
to change their majors if desired. Therefore, while placement as the criterion for success is 
easily accepted for JTPA and ETP progiams, performance contracts for vocational educa- 
tion would have to have multiple criteria, and what these criteria should be and how they 
should be weighted would be hotly debated. 

Second, the target populations are different. While the JTPA and ETP serve differ- 
ent populations, both groups are fairly homogeneous. JTPA serves only low-income 
youth and adults. The ETP program serves only individuals who have held jobs or who 
are in danger of losing them. In addition, contractors can be selective about who they 
admit to training. Postsecondary vocational education, on the other hand, serves a much 
more heterogeneous population, including a large number of smdents who enter directly 
from high school and have never held a job. Most postsecondary vocational educational 
institutions are committed to providing access to all students who have a chance of succeed- 
ing, and make special efforts to include the disadvantaged and other students with special 
needs. Performance contracting, at least as formulated for ETP, would provide a disincen- 
tive to enroll the disadvantaged and might lead to creaming. 
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Third, employers (and sometimes unions) work directly with ETP staff to select 
trainers, trainees, curriculum, method of training, and standards for successful completion. 
In vocational education programs, employers are normally involved only as members of an 
advisory committee that meets a few times a year. In addition, vocational education cur- 
ricula are not usually designed to meet the needs of individual employers. Indeed, many 
programs resist catering to the needs of individual employers and try instead to prepare stu- 
dents for a more general labor market 

Despite these differences, there are some important lessons to be learned from the 
experiences of the JTPA and ETP, and there do seem to be some opponunities for at least 
limited use of perfonnance contracting in postsecondary vocational education. One possi- 
bility would be not to make all vocational programs subject to performance contracting, but 
to limit it to certain ones that lend themselves to placement as the criterion for success. For 
example, vocational education programs that lead to state licenses may be good candidates 
for performance contracting. Similarly, some vocational education programs in community 
colleges such as office automation and computer assisted design are sometimes aimed pri- 
marily at retraining, and these may also be appropriate for peifomiance contracting. 

While it would not be appropriate to build performance contracts around placement 
alone in vocational education, it is certainly possible to envision other types of performance 
contracts. Possibilities include contracts with multiple measures of perfonnance, partial 
payments for meeting certain levels of achievement, or increased payments for serving dis- 
advantaged smd^-nts. 

Performance contracting could also be implemented so that an institution's perfor- 
mance is measured rather than a single program's. Such an approach would make it feasi- 
ble for instimtions to undertake risky programs along with ones that are highly likely to be 
successful. 

Performance-Based Funds Allocation Formulas 

In recent years, mathematical formulas have become increasingly popular for dis- 
tributing general purpose and categorical funds to eligible recipients. The Carl D. Perkins 
Vocational Education Act specifies the formula to be used to allocate the handicapped and 
disadvantaged setasides. While it does not require funds affected by other parts of the leg- 
islation to be distributed by formula, many states have adopted formulas of their own. 



Additionally, a few states use formulas to allocate state categorical aid for vocational 
education. 

These formulas typically distribute funds based on various measures of program 
inputs. Only a very few states include output measures in their funds distribution formu- 
las. Florida, for example, has established minimum placement standards for continued 
funding. Secondary vocational education programs that do not place at least seventy per- 
cent of their students are not eligible for refunding. Tennessee bases five percent of its 
state aid for higher education on Six measures of performance. Connecticut recenUy be- 
came the first state to allot general aid to school districts based pardy on deficiencies in stu- 
dent test scores (Rothman, 1987). Unlike performance-based funding in Tennessee, 
Connecticut's program directs more money to those with the lowest scores. 

To learn more about acmal experiences with performance-based allocation syrtems, 
we examined the history of Tennessee's postsecondary perfortr 'unding system. This 
system, which began in 1979 and had been in the planning stagv. *^^or almost seven years 
prior to that, is by far the oldest performance-based funds distribution system in education. 
Thus, with almost ten years of experience in operating this system, Tennessee seemed a 
likely source of useful information on ^e prospects of adapting such an approach to voca- 
tional education. 

Performance-Based Funding in Higher Education in Tennessee 

In 1979, The Tennessee Higher Education Commission (THEC) began implement- 
ing a new approach to funding the state's public colleges and universities. At the outset, 
performance funding was limited to two percent of general expenditures, but, henceforth, 
up to five percent of an institution's annual state allocation for instruction would be 
awarded based on performance. Tennessee has adopted performance standards in six 
areas: (1) program accreditation, (2) major filed assessment, (3) undergraduate general 
education outcomes, (4) alumni satisfaction surveys, (5) corrective measures, and (6) 
development and piloting of assessment instruments.^ 



^The number, content, and score of the standards have changed over time. Initially, performance funding 
standards were somewhat more process-oriented, encouraging institutions to implement the planning and 
assessment procedures that would make performance funding possible. As these procedures were 
implemented, standards became more outcome-oriented. For an excellent description of the history of 
perfonnance funding in Tennessee, sec Trudy W. Banta and Homer S. Fisher, "Performance Funding: 
Tennessee's Expcnment," in J. Folger (Ed.), Financial Incentives for Academic Quality: New Directions 
for Higher Education, No. 48. San Francisco. CA: Jossey-Bass, December, 1984, pp. 29-41. 
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In July, 1988, we visited the state capital and three local colleges and universities. 
We wanted to know more about how the system worked, what kinds of problems had been 
encountered, what solutions had been attempted, and how state and local personnel per- 
ceived the effectiveness of the performance-based funding. 

In Tennessee, higher education institutions may earn a supplement to their annual 
budget of up to five percent of their regular operating budget, based on how they score on 
standards developed in six areas of institutional performance. Out of a possible total of 
one-hundred and ten points, each institution may earn twenty points for program accredita- 
tion, thirty points for major field assessment, twenty points for undergraduate general edu- 
cation outcomes, fifteen points for alumni satisfaction surveys, fifteen points for corrective 
measures, and ten points for developing and piloting assessment instruments. An institu- 
tion scoring a full one-hundred and ten points would receive the entire five percent supple- 
ment. The supplement is prorated for institutions scoring fewer than one-hundred and ten 
points. For example, a score of eighty would entitie an institution to 72.7 percent (80 
110 = .727) of the five percent supplement; a score of fifty-five would earn fifty percent 
(55 + 110 = .50) of the five percent supplement. In practice, only a few institutions have 
received the full five percent, and all have received something. 

Although participation in the system is voluntary, all eligible public universities, 
community colleges, and technical institutes in Tennessee participate. To not participate 
would invite embarrassing questions from local governing boards, as well as the press and 
public at large. Hence, institutions dissatisfied with the system channel their energies into 
modifying the performance standards in ways they deem more appropriate, rather than 
opting out altogether. 

THEC is responsible for developing the performance standards and docs so in con- 
sultation with local institutions. Local institutions enjoy some discretion in implementing 
certain aspects of the standards and performance-measurement process, but local adminis- 
trative decisions of this type require approval by the local governing board. In some in- 
stances, approval be the state board or the THEC staff is required. Additionally, the 
Executive Director of THEC may, for good and reasonable cause and with concurrence of 
the state governing board, authorize modifications to the standards. Such modifications, 
however, must be applied uniformly to all institutions. 



25 rv 



Institutions receive a score in each of the six standard areas. The procedures for 
computing this score are in some instances simple and quick. In some areas, however, the 
procedures are tedious and differ for different types of postsecondary institutions. 

Scores for program accreditation are easy to det«?nmne. To qualify for a minimum 
score of ten points, at least sixty-five percent of an institution's programs eligible for ac- 
creditation must in fact be accredited. An institution will earn thirteen points if from sev- 
enty-four to eighty-one percent of its eligible programs arc accredited, fifteen points if from 
eighty-two to eighty-five percent are accredited, and an additional point for each additional 
three percent gain in the percentage of eligible programs accredited. 

In contrast to the procedures for determining accreditation points, the procedures 
for scoring the major field assessment are quite complex. Procedures are different for uni- 
versities and two-year institutions. Of a maximum of thirty points for this standard, a 
maximum of twenty points may be earned based on the performance of graduating students 
on approved undergraduate major Held tests, and a maximum of ten points may be earned 
based on an external review of masters' programs. Scores for perfomiancc of graduating 
students are based both on the percentage of students scoring above the national mean and 
on the extent of improvement over previous smdent performance. The external review of 
masters' programs is performed by a qualified consultant, approved by the THEC staff 
who determines whether or not the institution has met objective standards established by 
the Tennessee Council of Graduate Schools in each of ten categories, including screening 
and supervision of students, core cuniculum, comprehensive examination, and research. 

For the two-year colleges, scoring for the major field standard carries a maximum 
of ten points for smdent performance relative to the national mean and the extent of im- 
provement over the previous year's scores in programs leading to a license or certificate; 
ten points for similar performance in other major fields; and ten points for placement With 
respect to placement, a program is considered successful if it achieves a placement rate of 
seventy-five percent or better in fields related to training. To receive any points under this 
standard, at least sixty percent of a two-year institution's programs must have achieved this 
seventy -five percent placement rate, in which case five points are awarded. Institutions 
with from 70 to 79.9 percent of its programs achieving this rate receive seven points; from 
80 to 89.9 percent, nine points; and ten points when over ninety percent of its prograr»" 
achieve this seventy-five percent placement rate. 
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The standards for undergraduate general education outcomes award a maxinjum of 
twenty points based on an institution's mean value-added between entry and exit scores on 
the ACT COMP test and on the mean score of its students relative to national norms. All 
baccalaureate and associate level students are tested. If the number of students tested ex- 
ceeds fifteen-hundred, the institution may test a statistically representative sample of fifteen- 
hundred students. 

The maximum of fifteen points awarded for alumni satisfaction is based on surveys 
conducted every other year of all undergraduate alumni who graduated two years before the 
year in which the survey is conducted. Institutions are required to use an "evaluative sur- 
vey,*' which is defined as one yielding quantifiable indices of satisfaction with instructional 
programs and academic support services. Institutions are free to design their own survey 
instruments, but these must be approved by THEC staff and must remain in force for three 
successive surveys. Points are automatically deducted for response rates of less than thirty 
percent 

The fifth set of standards awards a maximum of fifteen points for evidence that an 
institution is taking corrective measures to address deficiencies in performance on the sec- 
ond, third, and fourth standard areas. Finally, the sixth standard area awards ten points as 
an incentive to an instimtion or group of institutions to develop and pilot tests and other 
assessment tools that have not yet been approved for use in the performance funding sys- 
tem. Five points are awarded for development and five points for piloting and testing. 

In sum, Tennessee's performance standards employ measures that vary consider- 
ably in their degree of objectivity, ease of measurement, and cost of collecting, maintain- 
ing, and reporting. The regulations explaining the procedures and definitions of relevant 
terms, while relatively easy to follow, nevertheless fill thirty-three single spaced pages. 
Considerable effort by both state and local staff has been devoted to modifying and refining 
these provisions over the state's nine year history with performance-based funding. 

From 1980 to 1988, Tennessee has provided an additional eighty-four million dol- 
lars to its public higher education institutions. Additionally, considerable staff time at both 
the state and local levels has been devoted to designing, implementing, and refining the 
funding system. What evidence is there that all of this effon has had any impact? 

Several local officials indicated to us that they were initially skeptical about the de- 
sirability of the system. Performance funding was seen as an irritant that many local ad- 



ministrators and fiu;ulty wished would go away. As it became apparent that the state was 
serious about continuing to implement the program, and as the comparative indicators of 
individual institutional performance became more widely known through the press, local 
institutions began to take performance funding more seriously. Enrollment declines at one 
of the lowest scoring institutions, for example, forced this campus to look seriously at what 
it was doing. 

The local officials we interviewed all reponed positive effects. One community 
college reported that peifonnance funding forced it, for the first time, to systematically ex- 
amine what happened to its students after graduation and to find out how they assessed 
their educational experience at the college. As the result of student follow-up prompted by 
performance-based funding, this college devoted greater resources to student services, 
especially financial aid administration. It inspired efforts to seek additional community 
support for offering "honor scholarships" to attract more top students and keep them en- 
rolled until graduation. The college also undertook a revamping of its allied health curricu- 
lum and testing. It cut down its student/teacher ratio and introduced new health registry ex- 
ams. As a result, the rate of attrition dropped fifty percent in its nursing program. 

At one of the state universities, performance funding led to an institutional self-ex- 
amination that identified needs to improve admissions, advising, and student services. It 
sparked a reassessment of the curriculum, producing a new experimental core curriculum in 
which the performance of students was systematically evaluated with that of students in the 
regular curriculum. The college also accelerated efforts to increase its accreditation 
performance. 

Another university also reported making major changes in advising and other stu- 
dent services as a result of student follow-up surveys required by performance-based 
funding. It greatiy increased its efforts to assess the capabilities of the students at the time 
of admission and their progress over time. 

Statewide, officials cite three major improvements in higher education: (1) a 
marked increase in the percentage of accreditable programs that are actually accredited; (2) 
an increase in tiie percentage of students in licensed and certified fields passing their pro- 
fessional examinations with scores exceeding the national norm; and (3) a steady rise in 
COMP scores. While it is impossible to unequivocally attribute these results to perfor- 
mance funding, there are no other obvious explanations. A conventional alternative expla- 
nation, that is, creaming, seems unlikely. As one local administrator told us when we 
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inquired about incentives to cream, "When ninety-five percent of my money is still driven 
by FTE and only five percent by performance, why would I give up $95 to make $5? I 
will still take any student who wants to enroll.'* Because it affects a relatively small 
percentage of total funding, performance funding in Tennessee operates at the margins and 
involves much lower risks than the all or nothing consequences of performance 
contracting. Hence, there are no strong incentives to admit only students who are likely to 
perform well. 

Despite these positive signs, performance funding in Tennessee has evoked \u 
share of criticism. At the outset, there was widespread resistance and apprehension on the 
part of many local administrators and faculty members who maintained that educational 
outcomes were not definable and who viewed performance funding as creating unaccept- 
able degrees of state intervention in campus affairs. Publicly releasing instimtional scores 
worried many who feared, with some justification, that false and misleading comparisons 
would be made among institutions. Others argued that performance funding directed addi- 
tional resources to instimtions least in need of help. Still others maintained that standards 
could not be implemented fairly because data was not comparable across institutions and 
because differences among institutions such as the mix of accredited and non-accredited 
programs would bias performance scores. 

While state staff have resolutely stuck to performance funding, they also have lis- 
tened carefully to local criticism, actively sought input and feedback, and have responded 
with modifications to the system. Thus, complaints that the system placed too much em- 
phasis on accreditation led to a reduction in the number of points awarded for this standLid. 
Efforts have been made to reduce the paperwork associated with major field assessment 
and undergraduate general education outcomes. Annual requirements for surveying alumni 
were shifted to biennial requirements in response to concerns about the cost of compliance, 
and state staff have encouraged networking and other means for sharing assessment in- 
struments to reduce the costs of student testing. In short, THEC has viewed performance 
funding as a flexible system, subject to change and refinement as the need arises. There are 
no illusions that the system is perfect, but there are strong feelings that despite its imperfec- 
tions performance funding has improved higher education in Tennessee. 

Tennessee's experience illustrates that if expectations for performance funding are 
not set unrealistically high, important improvements can be accomplished. When perfor- 
mance funding first started in Tennessee, no institution had a systematic process for inter- 
nal assessment of institutional objectives and performance; now all do. Without question, 
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the system has focused attention on standards, assessment, and performance — a 
prerequisite for any institution or group of professionals seeking self-improvement and 
self-regulation. Performance funding, however, has its limits, especially if it affects only 
five percent or less of an institution's finances. For administrators who are aggressively 
seeking change and improvement, program funding provides useful leverage. However, 
program funding is not likely to induce change in those who remain steadfastly resistant. It 
is possible to comply with performance funding and obtain some additional funds without 
really improving perfonnance. Additionally, perfonnance funding inevitably leads to some 
reduction in local academic autonomy and an increase in bureaucracy as pressures mount 
for a more uniform system of definitions and measurement that will reassure all the 
participants that they are being treated fairly. 

Policy Implications 

Tennessee's experience with performance-based funding offers a number of impor- 
tant lessons for efforts to shape a more outcome-oriented approach to state policy for post- 
secondary vocational education. First, not only is it possible to fashion a performance 
funding system with a relatively small amount of money, but, also, the success of the sys- 
tem may well depend on performance incentive being relatively small. Performance- 
based funding, as practiced in Tennessee, was designed to improve performance at the 
margins, not radically transform the higher education system by introducing possibly un- 
desirable incentives. Thus, because the financing of postsecondary education remained 
largely enrollment driven, incentives to cream and otherwise skew operations artificially in 
response to perfora[iancc standards were greatiy reduced. Moreover, the fact that perfor- 
mance funding allocated supplemental money rather than reallocated existing dollars helped 
to reduce opposition and secure constructive cooperation. 

Second, a performance funding system is more likely to be successful if it does not 
attempt to bypass or ignore existing institutional arrangements. While still exercising 
strong leadership at the state level, Tennessee continuously involved local administrators 
and faculty in die design and modification of all aspects of performance funding. 
Additionally, local administrators and faculty members were given subsrantial discretion in 
determining how best to implement standards, and there were virmally no strings on how 
institutions could use the funds they earned through performance fiinding. Participation in 
performance funding was voluntary from the beginning. While pressures from peer insti- 
tutions, the press, and consumers would have made it difficult for any institution not to 
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participate, institutions were always free to opt out if they felt they were treated unfairly or 
that the required effort was not worth the extra money. 

Third, financial incentives that stop formally at the institutional level can be effec- 
tive. It is not necessary that the system establish specific procedures for using financial in- 
centives to motivate individual departments, programs, or personnel. Leaving institutions 
free to determine how best to use the rewards of performance funding to motivate particular 
individuals or groups of individuals worked in Tennessee, especially when there were local 
administrators and faculty members who were able to use performance funding as leverage 
for their own agendas for improvement 

Fourth, including measures of year-to-year improvement and value-added in per- 
formance funding eliminates the problem that such systems reward only the highest per- 
forming institutions that may be the least in need of additional resources. Low performing 
institutions could and did receive additional money under performance funding in 
Tennessee, but only if they did, in fact, show improvement Such an approach is probably 
superior to systems that simply target more resources on low performers, with no attention 
to whether the additional funding leads to increases in performance. 

Fifth, designing and operating performance funding systems requires a substantial 
commitment of resources over an extended period of time. Tennessee began developing its 
performance funding system about seven years before its initial implementation. Once im- 
plemented, about four years passed before most local institutions began to take perfor- 
mance funding seriously and more fully recognized its potential for improving their own 
operations. Even as the system was accepted and more widely recognized as beneficial, 
ongoing needs to evaluate and modify the system required ongoing staff time at both the 
state and local level. Because of the extended period of time required from design to im- 
plementation to acceptance and further nKxliHcation, continuity and strength of leadership 
figured prominently in the success of the system. In this respect performance funding 
benefited from the strong support of former Governor Lamar Alexander, who led major ef- 
forts to strengthen accountability at all levels of education and who, later, as President of 
the University of Tennessee, continued to champion the system. Additionally, leadership 
of THEC changed only once over the eighteen yean spanning development of performance 
funding to the present, and both executive directors were strong advocates of performance 
funding. 



Sixth, incorporating multiple measures of performance increases the credibility and 
acceptability of performance funding. By rewarding institutions for performance on six 
sets of standards, and by using multiple measures within several of these standards, 
Tennessee avoided the criticism that the system was biased in favor of instimtions that 
happened to do better on an arbitrarily narrow selection of measures. While opting for 
multiple measures inevitably increased the complexity of performance funding, as well as 
the costs of collecting and reporting data, it nevertheless better reflected the multiple goals 
of postsecondary education and the divergent opinions about how one should me&sure 
educational performance. 

Finally, although Tennessee sought to preserve local flexibility, centralized efforts 
at the state level to achieve some uniformity were necessary. When performance is linked 
to funding, even a relatively small amount, the procedures require a set of measures that all 
can regard as fair. It is impossible, therefore, to let everyone devise their own individual 
systems for measuring and rewarding performance. 



IMPROVING STATE-LEVEL INFORMATION ON PERFORMANCE 

While there is growing evidence that appropriate performance criteria can be de- 
fined, measured, and linked to funding, obtaining accurate, timely, inexpensive data has 
posed major problems. Employment training programs, because they work with smaller 
numbers of clients than vocational education and because they have limited follow-up to 
relatively short periods following progr^ completion, have successfully collected data on 
placement Large scale, longer term efforts at tracing the labor market outcomes of partici- 
pants in either postsecondary vocational education or employment training have not been 
successful. 

In California, as in most states, littie has been done to systematically collect infor- 
mation on competency-based learning outcomes, although information on completers of 
vocational education programs has been successfully collected at the secondary and post- 
secondary levels. Except for information on completers, obtaining data on learning out- 
comes will require a major developmental effort, well beyond the scope of this paper. 
There are, however, some possibilities for significantly and ratiicr inexpensively improving 
state data on labor market outcomes. 



The Condition of Follow-up Information 



To evaluate employment training and vocational education programs in terms of la- 
bor niaiket outcomes, one would ideally like to have data on many of these outcomes over 
an extended period of time (at least for a year after participadon in a program and preferably 
for Hve years or more). In practice, a large scale follow-up of the labor market experience 
of program participants has been riddled with problems. The attempt by the now defunct 
Vocational Education Data System (VEDS) to require annual universal follow-up of pro- 
gram completers in vocational education failed dismally. Intended to determine employ- 
ment stams six months after program completion (limited to placement data rather than in- 
formation on hourly wage rates or earnings), the VEDS follow-up effort, which relied 
mainly on mail surveys, typically produced information on no more than twenty-five per- 
cent of the completers, rendering the data useless. Moreover, even had response rates been 
acceptable, the follow-up effort would have yielded no infomiation on the longer term labor 
market experience of vocational educaticm students. 

Much better and more comprehensive follow-up information has been generated by 
several national longitudinal snidies such as the National Longitudinal Study of the Senior 
Class of 1972, NLS Youth, and High School and Beyond. These large national samples 
achieve unifonnity in definitions and measurement methodology, and they have obtained 
excellent response rates. They follow participants over many years, permitting assess- 
ments of the long term effects of participating in vocational education and employment 
training. As powerful as these longitudinal studies are, however, they suffer from several 
drawbacks. 

First, they are costiy and complicated. They can only be done infrequently (once 
about every eight yea^o Is the current norm) and require much technical expertise. While 
they can provide good national estimates of the aggregate import of participation in voca- 
tional education, the sample sizes are not large enough to make comparisons among differ- 
ent states, let alone among different institutions or programs within states. 
Programmatically, national differences can be analyzed for approximately twenty-five pro- 
grams with the largest enrollments, far short of the several hundred different types of voca- 
tional education programs that are operated at the postsecondary level. In short, the longi- 
tudinal studies are excellent sources of data for national policy studies and might even be 
effectively used for policy assessments in some of the largest states where sufficient sample 
size can be generated. They arc not, however, viable tools for ongoing accountability 
assessments of the effectiveness of programs at the substate or institutional level. Hence, 
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they cannot play a major role in performance- based policy, other than to infrequently vali- 
daic less comprehensive and shoner term approaches to obtaining information on labor 
market outcomes. 

One promising source of nearly universal information on some labor market out- 
comes is the data maintained by each state's unemployment insurance recordkeeping sys- 
tem. Several states have been experimenting with procedures that use social security num- 
bers to merge student record information on participation in vocational education programs 
with the labor market data collected quanerly by the unemployment insurance system. In 
California, the Employment Training Panel (ETP) has been using this data to verify place- 
ment and duration of employment While the particulars vary among states, generally all of 
these systems can provide information on whether a former student is employed (i.e., gen- 
eral placement), the industry in which the student is employed, and the student's quanerly 
earnings. Moreover, these individuals may be followed over time for as long as they are 
employed or, in the event of unemployment, for as long as they are receiving unemploy- 
ment insurance. 

These files offer several advantages over one-time mail follow-up surveys. First, 
they are inexpensive to use and will produce data on a larger percentage of student-: than 
that obtained from responses to mail surveys. Second, the data is more accurate. The in- 
formation is based on actual wages paid from employers' payroll records rather than the 
self-report of the employee. Third, samples need not be limited to program completers. It 
is relatively easy to analyze a wide variety of participation and completion panems. Fourtli, 
extended longitudinal analysis is possible at relatively modest cost, compared to the na- 
tional longitudinal su;-veys. While these features make the unemployment insurance data 
quite useful, the files do suffer from the following shortcomings: 

• incomplete coverage; 

• information on occupation is usually not available; 

• earnings information is not adjusted for part-time employment; and 

» requirements for individual student record data with social security numbers. 

Coverage is problematic for two reasons. First, students who move out of state to 
work following participation in vocational education programs will not be include in slate 
data files and, through the lack of any other information, they will appear not to be in the 
labor force. Consequently, estimates of placement will be biased downward. Second, 
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some classifications of workers are exempt from unemployment insurance reponing re- 
quirements, and to the extent that students go to work in these types of employment, they 
also will appear to be unemployed or not in the labor force. 

Although the extent of coverage varies somewhat from state to state, the unem- 
ployment insurance data typically excludes certain classifications of employees who are ex- 
empt from the unemployment insurance code in each state and who are therefore ineligible 
to receive unemployment benefits. Self-employed workers (including sole stockholders of 
private corporations, sole proprietors, unpaid family workers, and independent contractors) 
are the largest single group of exempt employees and account for about ten percent of total 
employment in most states. People in the military are also exempt, and in some states there 
are substantial numbers of military personnel who can be counted as a percentage of total 
state employment. A few states exclude certain classifications of agricultural workers. 
Other exempt categories include worlcers in churches and some nonprofit organizations, 
students working while enrolled in school, elected officials, domestics, and some real es- 
tate brokers. 

With the exception of self-employed individuals and military personnel, these ex- 
empt categories do not pose serious problems for follow-up. They typically include small 
numbers of workers and are not common occupations for students taking vocational educa- 
tion, especially at the postsecondary level."* Identifying students who work out of state, are 
self-employed, or who are enlisted in the military, however, is more important. One state 
which uses unemployment insurance files for follow-up conducts mail and telephone sur- 
veys to examine only those students who cannot be found in the unemployment insurance 
files. Since more than eighty percent of students employed soon after leaving school will 
show up in the in-state unemployment insurance data, the magnitude of a supplementary 
mail or telephone survey is considerably less than if follow-up efforts had to rely on these 
methods alone.^ Consequently, with fewer students to survey, the mail and telephone ef 
forts may be more persistent in attempting to contact former students. Moreover, as the 
unemployment insurance data gready increases the minimum number of students for whom 
data is available, problems caused by a lack of response to the mail and telephone surveys 
arc less severe. 



^In California, for example, the exempt caiegorics ouier than self-employment and military personnel 
accounted for about 1.5 percent of total employment in 1986, 

higher rate of initial matching could also be obtained if neighboring states agreed to pool their 
unemploymeni insurance data, so that students working n nearby stales would be identified. 
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A second drawback of the un mployment insurance data is that states have gener- 
ally been reluctant to require employers to identify the occupations in which their employ- 
ees work. Updating occupational information on a quarterly or annual basis would be quite 
burdensome for employers. The employer does identify the Standard Industrial 
Classification (SIC) code that best reflects the nature of the business, but SIC codes are not 
usually a satisfactory indicator of the type of occupation in which a former student is em- 
ployed Consequently, it is difficult to use these data to detemiine the extent to which em- 
ployment is related to a student's training. Although many states do annually survey em- 
ployers to obtain an occupational profile, this data is reponed in aggregated form by each 
employer. It is not possible to use these surveys to link a particular individual to a particu- 
lar occupation. At best, one could use this aggregate data to determine a probability that, 
given employment in a particular SIC code, a person is employed in an occupation related 
to training. 

A possible solution to some of these difficulties would be to ask employers to indi- 
cate only at th of first hiring an employee the occupation in which the employee will 
initially be wc-A..»g. While such information would probably become outdated the longer 
an individual remained with the same firm, it would nevertheless provide useful informa- 
tion about relatedness to training at the point of entry. For policy purposes, this is the pri- 
mary concern. As the employer would have to supply the information only once for each 
employee and since the information would be supplied at the same time other data on the 
employee (name, social security number, and earnings) initially submitted, such a re- 
quirement would impose little additional reporting burden. 

A third shortcoming of the unemployment insurance data is that employers report 
only the total earnings for each employee during a quarter. Some measure of rate of pay 
(e.g., hourly wage or number of hours worked) is typically not included with the daia. 
Consequently, in comparing earnings of different individuals, it is impossible to control for 
differences resulting from part-time employment or employment commencing after the 
quarter has begun. As much part-time employment is voluntary, comparisons of program 
outcomes based solely on aggregate quarterly earnings will present a distorted picture. 

Although requiring employers to report an additional data element to permit correc- 
tions for the lack of comparability in quarterly earnings would increase data burden, such a 
requirenrtcnt would probably be much less burdensome than the requirement that they report 
occupational data. Such a variable could take one of two forms. First, employers could be 
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asked to report for each employee the total number of hours worked during a quarter. 
Second, they could be asked to report the hourly wage rate in effect during the last pay pe- 
riod of the quaner. As payroll systems, even for very small firms, are increasingly auto- 
mated (including production of the unemployment insurance quarterly reports), requiring 
one or the other of these items should not cause much additional inconvenience. Hours 
worked and hourly wages must be routinely maintained by the payroll system for part-time 
and hourly employees, and imputing an hourly wage rate for salaried employees could be 
made straightforward, even if somewhat arbitrary.^ Indeed, if such a requirement were 
limited to employers with automated payroll/reporting systems, wage rate data would prob- 
ably be available for well over three-fourths of employees in most states. 

The final difficulty with this approach to obtaining data on labor market outcomes is 
that it requires that student records be automated, based on an individual student record 
system, and that they contain the student's social security number for matching with 
unemployment insurance data files. All educational institutions must also be willing to 
provide this data to a central state agency, which can coordinate the matching with the state 
unemployment insurance files. In California, the community colleges do maintain, student 
record systems centrally in Sacramento, permitting the data to be merged with unemploy- 
ment insurance data. At the secondary level, however, there is no centralized student 
record data. 

In short, the unemployment insurance files provide a promising low-cost approach 
to obtaining current, acciu^te, and fairly comprehensive data on some selected labor market 
outcomes for students participating in vocational education programs. The approach is far 
from perfect and would benefit from a few modifications tiiat do not appear to be burden- 
some or expensive. Certainly, the information and coverage is far superior to the follow- 
up efforts attempted by VEDS. While this metiiod lacks the precision and detail of the 
larger national longitudinal smdies, it provides an economical alternative for obtaining insti- 
tutional and statewide data on an annual basis. Thus, potentially good data is available on 
labor market outcomes for participants in postsecondary vocational education and employ- 
ment training programs. 



^While salaried employees are often expected to work more than forty-hour workweeks, a forty-hour 
standard could be arbitrarily adopted for comparability purposes. 



CONCLUSION 



At the outset of this paper, we stated thai for performance-based policy to be effec- 
tive it must meet four requirements: it must be possible to define desired outcomes clearly; 
it must be possible to measure these outcomes accurately and efficiently; measures of per- 
formance must affect levels of funding; and useful inforaiation on outcomes must be avail- 
able for consumers and policytnakers. In the first four sections, we have demonstrated that 
it is possible to meet these four requirements. Defining and measuring program outcomes, 
linking them to funding, and obtaining information on outcomes for consumers and 
policymakers arc not merely abstract, untested, academic concepts. On the contrary, these 
practices are present in a wide variety of vocational education and employment training ac- 
tivities. What, then, are the implications for crafting a performance- based approach to vo- 
cational education and employment training policy? 

To begin, it is important to restate clearly the primary goal of adopting a perfor- 
mance-based orientation. Perfoiinance-based postsecondary vocational education and em- 
ployment training policy should have as its primary aim stimulating providers to increase 
the knowledge and skills of participants and to improve their prospects for successful par- 
ticipation in the labor market. Policy should seek to achieve this goal for all participants, as 
well as emphasize its realization for smdents with special needs. 

In seeking to realize this overarching goal, performance-based policy should con- 
tribute to a secondary objective — namely, focusing policy debates more on issues of out- 
comes and less on procedural matters. In some respects, the fact that performance- based 
policy creates a climate for asking hard questions about what postsecondary vocational ed- 
ucation and employment training should be seeking to accomplish and how best to deter- 
mine if these aims arc met is as important a.s whatever measurable char ges in performance 
can be attributed to a shift in state policy. Indeed, if state policy did nothing more than 
stimulate an ongoing attention to outcomes and to assessing institutional effectiveness on 
performance measures, it would have accomplished a great deal. 

Crafting such a policy will need to heed a number of important lessons that have 
been learned *o date. First, policy should allow localities substantial aiscretion to tailor per- 
formance-based policies to best fit their particular Circumstances. Localities vary greatly in 
how they organize and deliver postsecondary vocational education and errployment train- 
ing. Programmatic emphases also differ, reflecting variance in regional and local labor 
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markets and the types of students to be served. Seeking to impose a unitomi, highly spe- 
cific approach to implementing perforaiance-based policy is not likely to succeed. 

Second, state policy should require the adoption of multiple definitions of perfor- 
mance. The state has many options as to how it defines and measures performance criteria, 
but it should not rely on a single set of criteria (e g., labor market outcomes) or a single 
measure for such a set (e.g., placement rate). The state should develop measures of both 
labor market outcomes and educational outcomes and should adopt procedures for 
evaluating these measures for students with special needs, as well as for all other students. 
With respect to educational outcomes, the state should include some measures of value- 
added or changes in institutional performance over time so that performance is not mea- 
sured simply in terms of highest achievement, but also reflects improvement by students 
and institutions. In this way, gains in the performance of even relatively low achieving 
institutions can be recognized, encouraged, and rewarded. 

Third, in linking funding to performance, the state needs to consider carefully the 
differences between employment training and vocational education programs. Performanct 
contracting is a more suitable funding mechanism for employment training, while perfor- 
mance funding forrrulas that remain largely dnven by j.tudent FTE are more appropriate for 
postsecondary vocational education programs. In both types of mechanisms, the state 
should make provisions for adjusting for major factors that affect outcomes but are outside 
the direct control of vocational educaiuis. rwo of the most important of these factors are 
local economic conditions and student characteristics. In assessing the performance of 
employment training programs and postsecondary institutions, it is important that the com- 
parisons be made among like programs and like institutions, which face more or less equal 
local economic conditions. Fine tuning the procedures for ensuring such controls will take 
time and may require some research. Research may also be necessary to help the state ad- 
dress the difficult questions about how to treat unlikes. Thus, there may be a consensus 
that the same placement rate should not be expected of an institution confronting a local 
unemployment rate of ten percent as would be expected of one facing a rate of five percent. 
How much lower, however, may ihe placement rate be for the institution dealing with 
higher unemplojinent? 

In short, it is possible to define desirable performance objectives for postsecondary 
voc auonal education and employment training programs, measure them with reasonable 
ease and efficiency, tie Uiem to funding, and repon results to consumers and policymakers. 
Without doubt, developing and implementing performance- based policies poses 



39 

^'3 



difficulties. We do not mean to gloss over the extent of start-up time and a number of other 
technical problems that must be addressed. These problems, however, are not unique to 
performance funding. Any well crafted state policy, whether performance-based or not, 
will require time to implement and refine. 

The great advantage of performance-based policies is that the problems encountered 
with implementation are problems based on issues of central concern to the effective opera- 
tion of programs. Performance funding creates the climate for encouraging important de- 
bates over what the objectives of programs should be, how these aims should be defined 
and measured, how well they are being realized, where the systems arc successful, and 
where they need improvement. Answers to such questions are essential for sound policy. 
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